Remove duplicates from a Json String in Java?

Two options I can think of right off the bat:

• Parse the string using wither regex or tokens, add each key-value pair to a hashmap, and in the end recreate your JSON document with the duplicates removed. In this case though I would only remove key-value pairs that are exactly the same.
• Download the source code for org.json.JSONObject , and make a slight modification to the code to automatically leave out duplicates. This is a bit dangerous though. Another option is to create a modified version that simply validates and modifies.

Extending JSONObject Working Example

The below code allows you to create a JSONOBbject with a string containing duplicate keys. Exceptions are thrown only when you have two key-values that have the same key, but different values. This was because I think it would be a problem to choose at random which of the two should be assigned (e.g. the later value?). Of course this can be changed to work as you wish (e.g. keep last value for multiple keys).

Modified Class

import org.json.JSONException;
import org.json.JSONObject;


public class JSONObjectIgnoreDuplicates extends JSONObject {

     public JSONObjectIgnoreDuplicates(String json) {
        super(json);
    }

    public JSONObject putOnce(String key, Object value) throws JSONException {
            Object storedValue;
            if (key != null && value != null) {
                if ((storedValue = this.opt(key)) != null ) {
                    if(!storedValue.equals(value))                          //Only through Exception for different values with same key
                        throw new JSONException("Duplicate key \"" + key + "\"");
                    else
                        return this;
                }
                this.put(key, value);
            }
            return this;
        }
}

Main method

String json = "{\"Sign_In_Type\":\"Action\",\"Sign_In_Type\":\"Action\"}";
           try {
                JSONObject json_obj = new JSONObjectIgnoreDuplicates(json);
                String type = json_obj.getString("Sign_In_Type");
            } catch (JSONException e) {
                throw new RuntimeException(e);
            }   

Assuming that String json = "{\"Sign_In_Type\":\"Action\",\"Sign_In_Type\":\"Action\"}"; is a fiction for testing, can I ask whether creating the data as a String is the best choice in the first place? Why not a HashMap, or some other structure that either overwrites the subsequent reuses of a name or ignores them or throws an error when you add them? Don't wait until the conversion to JSON to make your data valid.

You can make use of the Jackson library to parse JSON. I'd problems doing the same task as you with org.json's package, but I turned to Jackson and I solved it: http://wiki.fasterxml.com/JacksonHome

I expanded Menelaos Bakopoulos answer, so that if inner values are also with duplicates, it won't create issues. the former solution worked on the first level only.

public class JSONObjectIgnoreDuplicates extends JSONObject {

    public JSONObjectIgnoreDuplicates(JSONTokener x) throws JSONException {
        super(x);
    }

    @Override
    public JSONObject putOnce(String key, Object value) throws JSONException {
        Object storedValue;
        if (key != null && value != null) {
            if ((storedValue = this.opt(key)) != null) {
                if (!storedValue.toString().equals(value.toString())) //Only throw Exception for different values with same key
                    throw new JSONException("Duplicate key \"" + key + "\"");
                else
                    return this;
            }
            this.put(key, value);
        }
        return this;
    }
}

private class JsonDupTokener extends JSONTokener {

    public JsonDupTokener(String s) {
        super(s);
    }

    @Override
    public Object nextValue() throws JSONException {
        char c = this.nextClean();
        switch (c) {
            case '\"':
            case '\'':
                return this.nextString(c);
            case '[':
                this.back();
                return new JSONArray(this);
            case '{':
                this.back();
                return new JSONObjectIgnoreDuplicates(this);
            default:
                StringBuffer sb;
                for (sb = new StringBuffer(); c >= 32 && ",:]}/\\\"[{;=#".indexOf(c) < 0; c = this.next()) {
                    sb.append(c);
                }

                this.back();
                String string = sb.toString().trim();
                if ("".equals(string)) {
                    throw this.syntaxError("Missing value");
                } else {
                    return JSONObject.stringToValue(string);
                }
        }
    }
}

Sorry I can't comment on Menelaos Bakopoulos' response due to reputation<50... Stupid system

Your solution unfortunately does not work here:

SEVERE: ERROR converting JSON to XML org.json.JSONException: Duplicate key "id"
org.json.JSONObject.putOnce(JSONObject.java:1076)
org.json.JSONObject.<init>(JSONObject.java:205)
org.json.JSONTokener.nextValue(JSONTokener.java:344)
org.json.JSONArray.<init>(JSONArray.java:125)
org.json.JSONTokener.nextValue(JSONTokener.java:348)
org.json.JSONObject.<init>(JSONObject.java:205)
JSONUtilities.JSONObjectIgnoreDuplicates.<init>(JSONUtilities.java:38)

It seems that calling super(json) in JSONObjectIgnoreDuplicates's constructor sends the code into a loop inside JSONObject, not JSONObjectIgnoreDuplicates ;{

I'm currently trying Asaf Bartov's solution, but there's no call from JSONObjectIgnoreDuplicates to JsonDupTokener, so appart from overloading the constructor of JSONObjectIgnoreDuplicates as follows, I don't see how it could work:

    public JSONObjectIgnoreDuplicates(String json) throws JSONException {
        this(new JSONDupTokener(json));
    }

EDIT: I can confirm this works :))))

Thanks everybody!!!!

With Google Gson you can decide what to do with duplicates in the input string. You need to register your own TypeAdapter responsible for serialization/deserialization of objects. It would look like this:

// this implementation converts the json string to a Map<String, String>,
// saving only the first duplicate key and dropping the rest
class NoDuplicatesAdapter extends TypeAdapter<HashMap<String, String>> {
    @Override
    public void write(JsonWriter out, HashMap<String, String> value) throws IOException {
        out.beginObject();
        for (Map.Entry<String, String> e: value.entrySet()) {
            out.name(e.getKey()).value(e.getValue());
        }
        out.endObject();
    }
    @Override
    public HashMap<String, String> read(JsonReader in) throws IOException {
        final HashMap<String, String> map = new HashMap<>();
        in.beginObject();
        while (in.hasNext()) {
            String name = in.nextName();
            // putting value to the map only if this key is not present;
            // here you can actually find duplicate keys and decide what to do with them
            map.putIfAbsent(name, in.nextString());
        }
        in.endObject();
        return map;
    }
}

Then you can parse your string:

String json = "{\"Sign_In_Type\":\"Action\",\"Sign_In_Type\":\"Action\"}";

Type mapType = new TypeToken<Map<String, String>>() {}.getType();

Map<String, String> map = new GsonBuilder()
        .registerTypeAdapter(mapType, new NoDuplicatesAdapter())
        .create()
        .fromJson(str, mapType);

The map will contain only the first "Sign_In_Type".