Creating a mock library

You have to separate the builder logic from the object to build. The method when has to return something which remembers the arguments, so that the invocation of thenReturn still knows the context.

For example

public class MyMock implements InvocationHandler {
    record Key(String name, List<?> arguments) {
        Key { // stream().toList() creates an immutable list allowing null
            arguments = arguments.stream().toList();
        }
        Key(String name, Object... arg) {
            this(name, arg == null? List.of(): Arrays.stream(arg).toList());
        }
    }
    final Map<Key, Function<Object[], Object>> rules = new HashMap<>();

    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        var rule = rules.get(new Key(method.getName(), args));
        if(rule == null) throw new IllegalStateException("No matching rule");
        return rule.apply(args);
    }
    public record Rule(MyMock mock, Key key) {
        public void thenReturn(Object val) {
            var existing = mock.rules.putIfAbsent(key, arg -> val);
            if(existing != null) throw new IllegalStateException("Rule already exist");
        }
        public void then(Function<Object[], Object> f) {
            var existing = mock.rules.putIfAbsent(key, Objects.requireNonNull(f));
            if(existing != null) throw new IllegalStateException("Rule already exist");
        }
    }
    public Rule when(String method, Object... args) {
        Key key = new Key(method, args);
        if(rules.containsKey(key)) throw new IllegalStateException("Rule already exist");
        return new Rule(this, key);
    }
}

This is already capable of executing your example literally, but also supports something like

MyMock m = new MyMock();
CalcInterface ref = (CalcInterface) Proxy.newProxyInstance(
        CalcInterface.class.getClassLoader(), new Class[]{CalcInterface.class}, m);

m.when("add", 1,2).thenReturn(3);
m.when("add", "x","y").thenReturn("xy");
AtomicInteger count = new AtomicInteger();
m.when("getValue").then(arg -> "getValue invoked " + count.incrementAndGet() + " times");

System.out.println(ref.add(1,2)); // prints 3
System.out.println(ref.add("x","y")); // prints "xy"
System.out.println(ref.getValue()); // prints getValue invoked 1 times
System.out.println(ref.getValue()); // prints getValue invoked 2 times

Note that when you want to add support for rules beyond simple value matching, a hash lookup will not work anymore. In that case you have to resort to a data structure you have to search linearly for a match.

The example above uses newer Java features like record classes but it shouldn’t be too hard to rewrite it for previous Java versions if required.

It’s also possible to redesign this code to use the real builder pattern, i.e. to use a builder to describe the configuration prior to creating the actual handler/mock instance. This allows the handler/mock to use an immutable state:

public class MyMock2 {
    public static Builder builder() {
        return new Builder();
    }
    public interface Rule {
        Builder thenReturn(Object val);
        Builder then(Function<Object[], Object> f);
    }
    public static class Builder {
        final Map<Key, Function<Object[], Object>> rules = new HashMap<>();

        public Rule when(String method, Object... args) {
            Key key = new Key(method, args);
            if(rules.containsKey(key))
                throw new IllegalStateException("Rule already exist");
            return new RuleImpl(this, key);
        }
        public <T> T build(Class<T> type) {
            Map<Key, Function<Object[], Object>> rules = Map.copyOf(this.rules);
            return type.cast(Proxy.newProxyInstance(type.getClassLoader(),
                new Class[]{ type }, (proxy, method, args) -> {
                   var rule = rules.get(new Key(method.getName(), args));
                   if(rule == null) throw new IllegalStateException("No matching rule");
                   return rule.apply(args);
                }));

        }
    }
    record RuleImpl(MyMock2.Builder builder, Key key) implements Rule {
        public Builder thenReturn(Object val) {
            var existing = builder.rules.putIfAbsent(key, arg -> val);
            if(existing != null) throw new IllegalStateException("Rule already exist");
            return builder;
        }
        public Builder then(Function<Object[], Object> f) {
            var existing = builder.rules.putIfAbsent(key, Objects.requireNonNull(f));
            if(existing != null) throw new IllegalStateException("Rule already exist");
            return builder;
        }
    }
    record Key(String name, List<?> arguments) {
        Key { // stream().toList() createns an immutable list allowing null
            arguments = arguments.stream().toList();
        }
        Key(String name, Object... arg) {
            this(name, arg == null? List.of(): Arrays.stream(arg).toList());
        }
    }
}

which can be used like

AtomicInteger count = new AtomicInteger();
CalcInterface ref = MyMock2.builder()
        .when("add", 1,2).thenReturn(3)
        .when("add", "x","y").thenReturn("xy")
        .when("getValue")
            .then(arg -> "getValue invoked " + count.incrementAndGet() + " times")
        .build(CalcInterface.class);

System.out.println(ref.add(1,2)); // prints 3
System.out.println(ref.add("x","y")); // prints "xy"
System.out.println(ref.getValue()); // prints getValue invoked 1 times
System.out.println(ref.getValue()); // prints getValue invoked 2 times