Dave Dave - 28 days ago 10
C# Question

How to send group separator (non printable ascii characters) with RestSharp RestClient from C#

Updated

I have completely reworked this question and included a complete working example.

My succinct question is now:
How come I'm not seeing individual characters of string "\u001d" when I use RestSharp RestClient and pick json as the RequestFormat options to send a simple object to a server? I send the output to a test server, and see only 1d when I examine the output with a binary editor.
I would expect

5C 75 30 30 31 64
('\','u','0','0','1','d') AND that is what you see if you just use NewtonSoft.Json to serialize the same simple object containing the string "\u001d". My understanding is that RestSharp will serialize your object as .json (if you pick the options accordingly) and send to server for deserialization.

using Newtonsoft.Json;
using RestSharp;
using RestSharp.Serializers;
using System;
using System.Collections.Generic;
using System.IO;

/// <summary>
/// To Test this,
/// 1. I first serialize a string to JSON and then print out the JSON string as a character array to show that
/// '\' 'u' '0' '0' '1' 'd' all show up in the serialized json string. I also put in original string \\001d (double \) for use later with RestSharp.
/// 2. I expect this json string to show up if I use RestSharp to send an object to a web service because RestSharp serializes everything to JSON
/// if it's so configured.
/// I make use of http://posttestserver.com/ to
/// act as "server", so I can just examine the data on that site. I copy the data on the server to a text file and open with binary editor.
/// Notice that there is just a '1d' present between 12 and 34.
/// You can perhaps duplicate what you get with just serialization by starting with (double \). But why should you need to do that?
/// Notice I try both RestClient's serializer and try to use NewtonSoft Serializer w/ RestClient.
/// 3. Finally I tried to send the serialized string with a client that had not been set up for JSON. Unfortunately, on server, it just shows up
/// as <String />. Not sure if it's me setting up client incorrectly or what's going on.
/// </summary>
class Program
{
static void Main(string[] args)
{
Tank tank1 = new Tank();
string string1 = "12\u001d34" + " " + "56\\u001d78" ; // you don't need double \\ with just serialization, but perhaps you need them with RestClient?
tank1.Description = string1;// new string(array1);
tank1.Id = 1;

// Show that we can serialize and each character of \u001d shows up as
JsonSerializerSettings settings = new JsonSerializerSettings { ReferenceLoopHandling = ReferenceLoopHandling.Ignore };
string conversion1 = JsonConvert.SerializeObject(tank1, Formatting.Indented, settings);
Console.WriteLine("JSON serialized string (showing hidden characters");

foreach(char dummyChar in conversion1.ToCharArray())
{
Console.Write(dummyChar + " ");
//Console.Write(conversion1.ToCharArray()[i] + " ");
} Console.WriteLine();

// Demonstrate that straight RestClient doesn't work.
RestClient client1 = new RestClient("http://posttestserver.com/"); // a website that let's you push data at it.

var request = new RestRequest(Method.POST); // method is Method.POST
request.AddHeader("Content-Type", "application/json");
request.AddHeader("X-ACME-API-VERSION", "1");
request.Resource = "post.php?dir=David"; // Put in unique name that you can easily find at http://posttestserver.com/data/2016/
request.RequestFormat = DataFormat.Json;

request.AddBody(tank1);

var response = client1.Execute(request); // after this line, go examine http://posttestserver.com/data/2016/ and find your post
// copy text to text file and open with binary editor. I claim you will see a 1d but not / (47) or u (117) or 0 (48) i.e. just 1d but not \u001d

// now try RequestClient w/ json serializer....
request.JsonSerializer = new JsonSerializerNewtonSoft();

response = client1.Execute(request);

// Finally, try just sending the json serialized stuff with a RestClient that has NOT been set to json stufff
RestClient client3 = new RestClient("http://posttestserver.com/"); // a website that let's you push data at it.
request.Resource = "post.php?dir=David"; // Put in unique name that you can find at http://posttestserver.com/data/2016/

var request3 = new RestRequest(Method.PUT); // method is Method.PUT // not sure what to put here
//request3.AddHeader("Content-Type", "application/json"); // not sure what to use here if anything
request3.AddHeader("X-ACME-API-VERSION", "1");
request3.Resource = "post.php?dir=David"; // Put in unique name that you can find at http://posttestserver.com/data/2016/
request3.RequestFormat = DataFormat.Xml; // not sure what to use here

request3.AddBody(conversion1);

var response3 = client3.Execute(request3); // hard to evaluate. Shows up at test server as <String />
}
}

interface ITank
{
int Id { get; set; }
string Description { get; set; }
}
public class Tank : ITank
{
public Tank() { }
public string Description { get; set; }
public int Id { get; set; }
}

public class Tanks
{
public Tanks() { }
public IEnumerable<Tank> Group { get; set; }
}

public class JsonSerializerNewtonSoft : ISerializer
{
private readonly Newtonsoft.Json.JsonSerializer _serializer;

/// <summary>
/// Default serializer
/// </summary>
public JsonSerializerNewtonSoft()
{
ContentType = "application/json";

_serializer = new Newtonsoft.Json.JsonSerializer
{
MissingMemberHandling = MissingMemberHandling.Ignore,
NullValueHandling = NullValueHandling.Include,
DefaultValueHandling = DefaultValueHandling.Include
};
}

/// <summary>
/// Default serializer with overload for allowing custom Json.NET settings
/// </summary>
public JsonSerializerNewtonSoft(Newtonsoft.Json.JsonSerializer serializer)
{
ContentType = "application/json";
_serializer = serializer;
}

/// <summary>
/// Serialize the object as JSON
/// </summary>
/// <param name="obj">Object to serialize</param>
/// <returns>JSON as String</returns>
public string Serialize(object obj)
{
using (var stringWriter = new StringWriter())
{
using (var jsonTextWriter = new JsonTextWriter(stringWriter))
{
jsonTextWriter.Formatting = Formatting.Indented;
jsonTextWriter.QuoteChar = '"';

_serializer.Serialize(jsonTextWriter, obj);
var result = stringWriter.ToString();
return result;
}
}
}

/// <summary>
/// Unused for JSON Serialization
/// </summary>
public string DateFormat { get; set; }

/// <summary>
/// Unused for JSON Serialization
/// </summary>
public string RootElement { get; set; }

/// <summary>
/// Unused for JSON Serialization
/// </summary>
public string Namespace { get; set; }

/// <summary>
/// Content type for serialized content
/// </summary>
public string ContentType { get; set; }
}

Answer

First, thank you for posting a Minimal, Complete, and Verifiable example which can be used to reproduce the problem; that made it much easier to help.

OK, there are a few things happening here which are leading to the results you are seeing. Let's look at them one at a time.


First, you are creating a string like this:

string string1 = "12\u001d34" + "  " + "56\\u001d78";

The number of backslashes you use definitely does matter because it has the same special meaning in C# as it does in JSON. Specifically, the notation \uxxxx in C# means, "insert the Unicode character with the 4-hexadecimal-digit (UTF-16) character code xxxx into the string". Conversely, the notation \\ means "insert a single \ character into the string". So, in the first part of your string, you are inserting the single character 0x001d, which is the Group Separator control character. In the second part of the string, you are inserting six characters: \, u, 0, 0, 1 and d. You can see the difference for yourself with a simple test program which dumps out the characters as hex digits:

public class Program
{
    public static void Main()
    {
        DumpCharsAsHex("\u001d");   // inserts actual 0x001d character (Group Separator) into string
        DumpCharsAsHex("\\u001d");  // inserts chars '\', 'u', '0', '0', '1', 'd' into string
    }

    private static void DumpCharsAsHex(string s)
    {
        if (s != null)
        {
            for (int i = 0; i < s.Length; i++)
            {
                int c = s[i];
                Console.Write(c.ToString("X") + " ");
            }
        }
        Console.WriteLine();
    }
}

Output:

1D 
5C 75 30 30 31 64

Fiddle: https://dotnetfiddle.net/8tjIiX


Second, there is definitely different behavior between Json.Net and SimpleJson (the RestSharp internal serializer) with respect to control characters that are embedded in a string. Json.Net recognizes control characters and converts them into their proper JSON escape sequence. (This conversion is done by the JavaScriptUtils.WriteEscapedJavaScriptString method, which in turn calls StringUtils.ToCharAsUnicode.) RestSharp, on the other hand, does no such conversion, and just passes the invisible control character through the JSON unchanged. (You can see this in SimpleJson.EscapeToJavascriptString.)
Again, a simple test program demonstrates the difference:

public class Program
{
    public static void Main()
    {
        Foo foo = new Foo { Bar = "\u001d" };

        string json = Newtonsoft.Json.JsonConvert.SerializeObject(foo);
        Console.WriteLine(json);
        DumpCharsAsHex(json);

        string json2 = RestSharp.SimpleJson.SerializeObject(foo);
        Console.WriteLine(json2);
        DumpCharsAsHex(json2);
    }

    private static void DumpCharsAsHex(string s)
    {
        if (s != null)
        {
            for (int i = 0; i < s.Length; i++)
            {
                int c = s[i];
                Console.Write(c.ToString("X") + " ");
            }
        }
        Console.WriteLine();
    }
}

public class Foo
{
    public string Bar { get; set; } 
}

Output:

{"Bar":"\u001d"}
7B 22 42 61 72 22 3A 22 5C 75 30 30 31 64 22 7D 
{"Bar":""}
7B 22 42 61 72 22 3A 22 1D 22 7D

Fiddle: https://dotnetfiddle.net/caxZfq

As you can see, in the first JSON, produced by Json.Net, the control character in the original string was converted into JSON character escape notation, which coincidentally looks just like the original C# code. When the JSON is deserialized on the other end, this will get converted back into a control character.

In the second JSON, produced by RestSharp, the control character is actually present (the 1D between the 22s) even though it is not visible in the JSON output. I should note that this is definitely incorrect behavior according to Section 9 of the JSON spec (emphasis mine):

A string is a sequence of Unicode code points wrapped with quotation marks (U+0022). All characters may be placed within the quotation marks except for the characters that must be escaped: quotation mark (U+0022), reverse solidus (U+005C), and the control characters U+0000 to U+001F.

Since the control character is left in, it could get mangled or reinterpretted in an undesirable way during transmission over the wire or during deserialization on the other end.


Third, in your code it looks like you are trying to use Json.Net as a substitute serializer for RestSharp, but it doesn't seem to be making a difference in your results. The reason why is because you have your statements out of order.

You are doing this:

var request = new RestRequest(Method.POST);
...
request.RequestFormat = DataFormat.Json;
request.AddBody(tank1);
request.JsonSerializer = new JsonSerializerNewtonSoft();
response = client1.Execute(request);

Notice that you are calling AddBody before you set the JsonSerializer on the request. RestRequest.AddBody is the method that calls the serializer to get the JSON and adds the result to the body of the request; this is not done by RestClient.Execute. So by the time you set the alternate JSON serializer, it's too late-- you've already used the internal serializer to add the JSON to the body of the request and the alternate serializer is never called. Reverse the order of those two statements and it should work the way you want.

var request = new RestRequest(Method.POST);
...
request.RequestFormat = DataFormat.Json;
request.JsonSerializer = new JsonSerializerNewtonSoft();
request.AddBody(tank1);
response = client1.Execute(request);

Hope this makes sense.