Vision-Language Models (VLMs), with their strong reasoning and planning capabilities, are widely used in embodied decision-making (EDM) tasks such as autonomous driving and robotic manipulation. Recent research has increasingly explored adversarial attacks on VLMs to explore their vulnerabilities. However, these attacks either rely on overly strong assumptions, requiring full knowledge of the victim VLM, which is impractical for attacking VLM-based EDM systems, or exhibit limited effectiveness. The latter disrupts the system’s perception of most semantics in the image, thus interrupting the reasoning process of the VLM due to the inconsistency between perception and the task context defined by system prompts, and resulting in invalid outputs that fail to influence interactions with the physical world.

To this end, we propose a fine-grained adversarial attack framework, AdvEDM, which leverages the vision-text encoder in VLM-based EDM systems as the surrogate model, as it is typically pre-trained and easy for the adversary to access. Specifically, AdvEDM modifies the VLM's perception of only a few key objects while preserving the semantics of the remaining regions. This attack significantly reduces conflicts with the task context defined by the system prompts, making VLMs output valid but incorrect decisions and affecting the actions of entities, thus leading to a more substantial safety threat in the physical world. We design two variants of based on this framework, AdvEDM-R and AdvEDM-A, which are used to respectively remove the semantics of a specific object from the image and add the semantics of a new object into the image. The experimental results in both general scenarios and EDM tasks indicate excellent fine-grained control and attack effectiveness.