The scalar field command does not do any calculation. It only shows the values, and the last values in the table are the ones shown.
To illustrate:
Now, add a repeated row and notice how the value in the field is last in the table.
Instead, you can use the Pivot command to group based on your x and y. Try selecting the columns and adding the Pivot command. This command has a place to select what to output. Here we show the count.
If you want the mean value, select that and then use the gear menu to Extract the calculated values.
Now, the mean values are in your table, and they can be input to the scalar.
In the future, we may add some calculations into the scalar, but this would be the approach to use for now. Essentially, you are creating a date of pipeline where the output from one command can be used as the input to another.