OData third look

ODataLogo-96This is a follow up on my OData first look and OData second look post. The sample code builds on the previous post and shows projection, data formats, batch processing and actions.
Every part has the exceptions and how I solved them.

Projection

Create a new object with the selected information from the OData resource. This way you create a project or a view on that data.
This one actually just worked without exceptions

Code

Select the SurveyName and the Reports from a Survey and store them in a View object.

public class View {
    public string NameOfSurvey { get; set; }
    public ICollection< Report> Reports { get; set; }
}
var projection = context.CreateQuery<Survey>("Surveys")
    .Select(s => new View { 
        NameOfSurvey = s.SurveyName, 
        Reports = s.Reports })
    .First();

Data format

The format of the data in the messages can be JSON or ATOM/XML and is configurable. The default is XML.

Exception

When you call the UseJson method without a parameter, you must use the LoadServiceModel property to provide a valid IEdmModel instance.

Solution

Load the metadata and parse it to an IEdmModel. Then set the Format to JSON. This is not needed when the default ATOM/XML is used.

Code

var model = default(Microsoft.Data.Edm.IEdmModel);
var metadataRequest = WebRequest.Create(context.GetMetadataUri());
using (var response = metadataRequest.GetResponse()) {
    using (var stream = response.GetResponseStream()) {
        using (var reader = System.Xml.XmlReader.Create(stream)) {
            model = Microsoft.Data.Edm.Csdl.EdmxReader.Parse(reader);
        }
    }
    response.Close();
}
context.Format.LoadServiceModel = () => model;
context.Format.UseJson();

Exception #2

415 Unsupported Media Type

Solution

The $count will return an integer and only that, not an object. JSON and ATOM/XML cannot handle this. You must read the bare response and parse it to an integer.

Code

// results in HTTP415
var requestNumberOfCompletedRecordsForFlightSurvey = new DataServiceRequest<int>(
    new Uri(context.BaseUri, "Surveys('1')/DatabaseReports/$count"));

// bare WebRequest will succeed
var requestNumberOfCompletedRecordsForFlightSurvey =
    new Uri(context.BaseUri, "Surveys('1')/DatabaseReports/$count?$filter=Status eq 'Complete'");
var request = WebRequest.Create(requestNumberOfCompletedRecordsForFlightSurvey);
using(var response = request.GetResponse()) {
    using(var stream = response.GetResponseStream()) {
        using(var reader = new System.IO.StreamReader(stream)) {
            var content = reader.ReadToEnd();
            int count = 0;
            if (int.TryParse(content, out count)) Console.WriteLine("Response '{0}'", count);
            else Console.WriteLine("Response not a number ({0})", content);
        }
    }
    response.Close();
}

Batch

Combine multiple requests as one message to the OData service. The result is again one message, containing all the results.

Nice bonus is that the context will “cache” expanded properties. The cache fills in the gaps for requests that did not expand.
In the code you’ll see that the second request expands on Reports and that only the first result misses the information. The third request expands on the DatabaseReports which were missing before. The last request gets all properties set without expanding.

Exception

415 Unsupported Media Type (again)

Solution

The exception is part of the complete result. Check for the result.StatusCode while iterating the batch.

Code

var requestFlightSurveyOperation = new DataServiceRequest<Survey>(
    new Uri(context.BaseUri, "FlightSurvey"));
var requestFlightSurvey = new DataServiceRequest<Survey>(
    new Uri(context.BaseUri, "Surveys?$filter=SurveyName eq 'Flight'&$expand=Reports"));
var requestFlightSurveyById = new DataServiceRequest<Survey>(
    new Uri(context.BaseUri, "Surveys('1')?$expand=DatabaseReports"));
var requestFlightSurveyOperationParameter = new DataServiceRequest<Survey>(
    new Uri(context.BaseUri, "GetSurvey?surveyName='Flight'" ));
// one request message for all requests with one response message containing all results
var batch = context.ExecuteBatch(requestFlightSurveyOperation, requestFlightSurvey, requestFlightSurveyById, requestFlightSurveyOperationParameter);
foreach (QueryOperationResponse result in batch) {
    if (result.StatusCode > 299 || result.StatusCode < 200) {
        Console.WriteLine( "An error occurred.");
        Console.WriteLine(result.Error.Message);
    } else if (result.Query.ElementType == typeof( Survey)) {
        foreach (Survey s in result) {
            Console.WriteLine("Found survey {0} with {1} reports and {2} database reports",
                s.SurveyName,
                s.Reports == null ? "null" : s.Reports.Count.ToString(),
                s.DatabaseReports == null ? "null" : s.DatabaseReports.Count.ToString());
        }
    }
}

Action

An action can mutate the data. This involves implementing some interfaces:

  • IServiceProvider, on the DataService to provide the IDataServiceActionProvider implementation
  • IDataServiceActionProvider implementation to be returned in the GetService of the DataService
  • IDataServiceInvokable implementation for the action
  • IUpdatable, on the DataProvider, just clear the default implementation of SaveChanges
  • IDataServiceUpdateProvider2, on the DataProvider, ScheduleInvokable should invoke the invokable

After implementing all this you can check it with the $select=DataProvider.* request
http://MACHINENAME/Reporting.Service/SurveyServiceOData.svc/Surveys?$select=SurveyProvider.*

Exception

‘The service operation ‘GetLastReport’ returned by the provider is not read-only. Please make sure that all the service operations are set to read-only.’

Solution

Make sure to call getLastReportAction.SetReadOnly();

Exception #2

‘The service action ‘GetLastReport’ has the binding parameter of type ‘Reporting.Entities.Survey’, but there is no visible resource set for that type.

Solution

Use the types from the IDataServiceMetadataProvider. Don’t try to create them yourself.

Code

var metadataprovider = context.GetService(typeof(IDataServiceMetadataProvider)) as IDataServiceMetadataProvider;
var surveyType = metadataprovider.Types.SingleOrDefault(s => s.Name == "Survey");

Exception #3

When ‘returnType’ is an entity type or an entity collection type, ‘resultSetPathExpression’ and ‘resultSet’ cannot be both null and the resource type of the result set must be assignable from ‘returnType’.

Solution

Use the right return resourceSet when creating the ServiceAction.

Exception #4

404 not found

Solution

Set breakpoint in every method of IDataServiceActionProvider and debug to the cause. Mine was the lookup statement in the TryResolveServiceAction.

Exception #5

405 Method Not Allowed

Solution

Do a POST not a GET

Exception #6

501 Not Implemented The data source must implement IUpdatable, IDataServiceUpdateProvider or IDataServiceUpdateProvider2 to support updates.

Solution

Implement the required interfaces on the DataProvider. (Data.SurveyProvider)

Exception #7

204 No content

Solution

The parameter can be a Query in stead of a single object. Use something like the First() statement to get the object.

Code

public class SurveyServiceOData : DataService<Data.SurveyProvider>, IServiceProvider {
    public static void InitializeService(DataServiceConfiguration config) {
        // other config left out for clarity
        config.SetServiceActionAccessRule("*", ServiceActionRights.Invoke);
    }
    // Other operations here ...

    public object GetService(Type serviceType) {
        if (serviceType == typeof(IDataServiceActionProvider)) {
            return new SurveyServiceActionProvider();
        } else {
            return null;
        }
    }
}
public class SurveyServiceActionProvider : IDataServiceActionProvider
{
    public bool AdvertiseServiceAction(DataServiceOperationContext operationContext, ServiceAction serviceAction, object resourceInstance, bool resourceInstanceInFeed, ref Microsoft.Data.OData.ODataAction actionToSerialize) {
        // everything is available
        return true;
    }
    public IDataServiceInvokable CreateInvokable(DataServiceOperationContext operationContext, ServiceAction serviceAction, object[] parameterTokens) {
        if (serviceAction.Name == "GetLastReport") return new GetLastReportActionInvokable(parameterTokens);
        throw new NotImplementedException();
    }
    public IEnumerable<ServiceAction> GetServiceActions(DataServiceOperationContext operationContext) {
        return GetActions(operationContext);
    }
    public IEnumerable<ServiceAction> GetServiceActionsByBindingParameterType(DataServiceOperationContext operationContext, ResourceType bindingParameterType) {
        var actions = GetActions(operationContext);
        return actions.Where(x => x.BindingParameter != null)
                      .Where(x => x.BindingParameter.ParameterType == bindingParameterType);
    }
    public bool TryResolveServiceAction(DataServiceOperationContext operationContext, string serviceActionName, out ServiceAction serviceAction) {
        var actions = GetActions(operationContext);
        serviceAction = actions.FirstOrDefault(x => x.Name.Equals(serviceActionName));
        return serviceAction != null;
    }
    private List<ServiceAction> GetActions(DataServiceOperationContext context) {
        var result = new List<ServiceAction>();
        // types from metadata
        var metadataprovider = context.GetService(typeof(IDataServiceMetadataProvider)) as IDataServiceMetadataProvider;
        var surveyType = metadataprovider.Types.SingleOrDefault(s => s.Name == "Survey");
        var reportType = metadataprovider.Types.SingleOrDefault(s => s.Name == "Report");
        var reportResourceSet = new ResourceSet("Reports", reportType);
        // service action
        var getLastReportAction = new ServiceAction(
            "GetLastReport",    // name of the action
            reportType,         // return type
            reportResourceSet,  // resource set of the return type
            OperationParameterBindingKind.Always,   // binding
            new[] {             // parameters, with first parameter the binding parameter
                new ServiceActionParameter("survey", surveyType)
            }
        );
        getLastReportAction.SetReadOnly();
        result.Add(getLastReportAction);
        return result;
    }
}
public class GetLastReportActionInvokable : IDataServiceInvokable {
    object[] _parameterTokens;
    Entities.Report result;
    public GetLastReportActionInvokable(object[] parameterTokens) {
        _parameterTokens = parameterTokens;
    }
    public object GetResult() {
        return result;
    }
    public void Invoke() {
        try {
            var parameter1 = _parameterTokens[0];
            var survey = parameter1 as Entities.Survey;
            if (parameter1 is EnumerableQuery) {
                survey = (parameter1 as IEnumerable<Entities.Survey>).First();
            }
            if (survey != null && survey.Reports != null) {
                result = survey.Reports.OrderByDescending(r => r.Created).FirstOrDefault();
            }
        } catch {
            throw new DataServiceException(500, "Exception executing action GetLastReport");
        }
    }
}
public class SurveyProvider : IUpdatable, IDataServiceUpdateProvider2 {
    // IQueryable properties here ...

    // other IUpdatable left at "throw new NotImplementedException()"
    // this makes the provider readonly as changes are not persisted
    public void SaveChanges() { }

    // other IDataServiceUpdateProvider2 left at 
    //       "throw new NotImplementedException()"
    public void ScheduleInvokable(IDataServiceInvokable invokable) {
        invokable.Invoke();
    }
}

Conclusion

Projections, batching and format are nice features out-of-the-box. Actions take a lot of implementation when you use your own DataProvider, but can be very powerful.
This concludes my exploration of OData. I think of it as a database service based on open standards for compatibility with some limitations and some extra’s. Choose this when you want users to explore your data from any platform, but leave it when you cannot work around the limited query language.

References

Using actions to implement server-side behavior on MSDN
How IDataServiceActionProvider works post on MSDN blog
ActionProvider example on codeplex

Posted in Development | Tagged , , , | 1 Comment

OData second look

ODataLogo-96This is a follow up on my OData first look post. Since then I’ve moved the service to IIS for better fiddler support. The sample code builds on the previous post and shows how to load related data in one call, the ability to overcome model changes and how to use operations.
Every part has the exceptions and how I solved them.

Expand

With the Expand operation the eager loading of related data is executed. This means that the property (set in the Expand parameter) will be supplied as an object in the response. Default a link would be returned for later retrieval of the object.

Exception

The property is returned in the response, but not available in the result object. With LoadProperty the property is set, but that is not why you call expand.

Solution

MergeOption of the context should be set. The default AppendOnly does not allow “changes” and ignores the expanded data. With NoTracking the LoadProperty will throw an exception, but the expand operation is executed and applied. PreserveChanges or OverwriteChanges support both Expand and LoadProperty, since my data is readonly both options are applicable.

Code

context.MergeOption = MergeOption.OverwriteChanges;
var lastReport = context.CreateQuery<Report>("Reports")
    .Expand("Survey")
    .Where(r => r.SurveyId == "1")
    .OrderByDescending(r => r.Created)
    .First();

Changes to the model

You can create a client for you odata service on three different ways

  1. Reference the entities assembly, update the client when the service changes
  2. Create a copy of the entities, your project is more standalone
  3. Generate a Service Reference, this combines 1 and 2, by generating the types from the entities assembly as a copy in the project

What type of client you use, changes to the model can break the application.

Exception

The property ‘x’ does not exist on type ‘y’. Make sure to only use property names that are defined by the type.

Solution

On the DataServiceContext is a property that instructs it to ignore missing properties. Whatever property cannot be mapped (renamed, removed or added) the value stays null or default.
Make note that changing the type of a property can cause an exception. A DateTime will fit in a String, but not every String will fit in a DateTime.

Code

var context = new DataServiceContext(uri, DataServiceProtocolVersion.V3);
context.IgnoreMissingProperties = true;

Operations

Operations can be used to handle the data. This offers more control to the developer. My sample does the filtering on the name. Maybe for security reasons or for performance.

Exception

I’ve got a 404 Not found exception on my first try.

Solution

The uri constructor replaces everything after the last slash. Make sure you end the baseUri with a ‘/’.

Code

// Dataservice:
public static void InitializeService(DataServiceConfiguration config) {
     // other config left out for clarity
     config.SetServiceOperationAccessRule("*", ServiceOperationRights.AllRead);
}

[WebGet]
public Reporting.Entities.Survey FlightSurvey() {
    return CurrentDataSource.Surveys
        .Where(x => x.SurveyName.Equals("flight"))
        .FirstOrDefault();
}

// client
var uri = new Uri ("http://MACHINENAME/Reporting.Service/SurveyServiceOData.svc/");
var context = new DataServiceContext(uri, DataServiceProtocolVersion.V3);
var flight = new Uri (uri, "FlightSurvey");
var survey = context.Execute<Survey>(flight).First();

Operations with parameter

Operations can have parameters. In my sample above the name could be a parameter.

Exception

Query options $select, $expand, $filter, $orderby, $inlinecount, $skip, $skiptoken and $top are not supported by this request method or cannot be applied to the requested resource.

Solution

Do a ToList() on the query before calling First(). This executes the odata query (without $top=1), reads the result in a list and then returns the first item.

Code

// DataService: 
[WebGet]
public Reporting.Entities.Survey GetSurvey(string surveyName) {
    return CurrentDataSource.Surveys
        .Where(x => x.SurveyName.Equals(surveyName))
        .FirstOrDefault();
}

// client
var uri = new Uri ("http://MACHINENAME/Reporting.Service/SurveyServiceOData.svc/");
var context = new DataServiceContext(uri, DataServiceProtocolVersion.V3);
var survey = context.CreateQuery<Survey >("GetSurvey" )
    .AddQueryOption("surveyName", "'flight'")
    .ToList().First();

Conclusion

With lazy loading and operations I can limit the traffic. Looking deeper into WCF data service, my use-case implementation is getting clearer.

References

Calling Service Operations article on MSDN
Framework source code of ClientType showing how IgnoreMissingProperties works. Exception or Ignore, no hooks for custom handling.

Posted in Development | Tagged , , , , | 1 Comment

Week roundup

Last week recap and links:
Image courtesy of kanate / FreeDigitalPhotos.net

Image courtesy of kanate / FreeDigitalPhotos.net

What are your best reads this week? Leave them in the comments below.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

First look at OData

ODataLogo-96I’ve heard about OData back in 2012 but never used it. Now I might have a use case for OData and want to explore it. Below are the parts of my sample WCF DataService project.
Every part has the exceptions and how I solved them.

Entities

My first project contains the entities of the sample project. A Survey and the Reports on it. Master-Detail with the surveyId stored in the Report.

Exception

On data context type ‘SurveyProvider’, there is a top IQueryable property ‘Surveys’ whose element type is not an entity type. Make sure that the IQueryable property is of entity type or specify the IgnoreProperties attribute on the data context type to ignore this property.

Solution

This exception is from the datacontext in the next part, but the cause is the missing [DataServiceKey(“Id”)] on the entities.

Code

[DataServiceKey("Id")]
public partial class Survey {
    public string Id { get; set; }
    public string SurveyName { get; set; }
    [InverseProperty("SurveyId")]
    public virtual ICollection<Report> Reports { get; set; }
}
[DataServiceKey("Id")]
public partial class Report {
    public string Id { get; set; }
    public DateTime Created { get; set; }
    public string SurveyId { get; set; }
    [ForeignKey("SurveyId")]
    public virtual Survey Survey { get; set; }
}

DataContext

The data project is for the reflection provider that holds the (test)data. All data is stored in lists.

Exception

HTTP 204 no content

Solution

Property is null in LoadProperty. Make sure the DataContext provides a value for the Master/Detail.

Code

public class SurveyProvider {
    private List<Survey> surveys = new List<Survey> {
        new Survey { Id = "1", SurveyName = "Flight" }
    };
    private List<Report> reports = new List<Report> { 
        new Report { Id = "1", SurveyId = "1", Created = new DateTime(2015,1,1,10,0,0) },
        new Report { Id = "2", SurveyId = "1", Created = new DateTime(2015,1,1,11,0,0) },
        new Report { Id = "3", SurveyId = "1", Created = new DateTime(2015,1,1,12,0,0) },
        new Report { Id = "4", SurveyId = "1", Created = new DateTime(2015,1,1,13,0,0) },
        new Report { Id = "5", SurveyId = "1", Created = new DateTime(2015,1,1,14,0,0) },
    };
    public SurveyProvider() {
        surveys.ForEach(s => s.Reports = reports.Where(r => r.SurveyId == s.Id).ToList());
        reports.ForEach(r => r.Survey = surveys.First(s => s.Id == r.SurveyId));
    }
    public IQueryable<Survey> Surveys {
        get { return surveys.AsQueryable(); }
    }
    public IQueryable<Report> Reports {
        get { return reports.AsQueryable(); }
    }
}

DataService

For easy debugging and trouble shooting enable the extra information on exceptions for the DataService. (See the code)

Also look at the metadata with the $metadata query option: http://localhost:53021/SurveyServiceOData.svc/$metadata

Host this in IIS to capture all traffic, because you’ll miss the lazy loading when hosting in IISExpress.

Code

[ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public class SurveyServiceOData : DataService<SurveyProvider> {
    public static void InitializeService(DataServiceConfiguration config) {
        config.UseVerboseErrors = true;
        config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
        config.DataServiceBehavior.MaxProtocolVersion = 
            DataServiceProtocolVersion.V3;
    }
}

Client

Exception

HTTP Error 400. The request hostname is invalid.
or no traffic captured with fiddler.

Solution

Use the localhost.fiddler when hosting in IISExpress.

var uri = new Uri("http://localhost.fiddler:53021/SurveyServiceOData.svc");

Exception

The remote name could not be resolved: ‘localhost.fiddler’.

Solution

Set the proxy to fiddler.

<configuration>
  <!-- other settings in app.config -->
  <system.net>
    <defaultProxy enabled="true" >
      <proxy proxyaddress="http://localhost:8888"/>
    </defaultProxy>
  </system.net>
</configuration>

Exception

The MaxDataServiceVersion ‘2.0’ is too low for the response. The lowest supported version is ‘3.0’.

Solution

Make sure the versions of the DataService and Client match. See also the InitializeService method of the DataService.

var ctx = new DataServiceContext(uri, DataServiceProtocolVersion.V3);

Exception

A property with name ‘Reports’ on type ‘Reporting.Entities.Survey’ has kind ‘Structural’, but it is expected to be of kind ‘Navigation’.

Solution

install-package EntityFramework

Code

static void Main(string[] args) {
    // use MachineName
    var uri = new Uri("http://localhost.fiddler:53021/SurveyServiceOData.svc");
    var context = new DataServiceContext(uri, DataServiceProtocolVersion.V3);
    var survey = context.CreateQuery<Survey>("Surveys").First();
    Console.WriteLine("Found survey {0} with {1} reports",
        survey.SurveyName, survey.Reports == null ? "null" : survey.Reports.Count.ToString());
    var response = context.LoadProperty(survey, "Reports");
    Console.WriteLine("LoadProperty Reports on Survey returned {0}", response.StatusCode);
    Console.WriteLine("Found survey {0} with {1} reports",
        survey.SurveyName, survey.Reports.Count.ToString());
    var pointInTime = new DateTime(2015, 1, 1, 12, 0, 0);
    var report = context.CreateQuery<Report>("Reports")
            .Where(r => r.SurveyId == "1")
            .Where(r => r.Created == pointInTime)
            .First();
    Console.WriteLine("Found report on {0} for Survey {1}",
            report.Created, report.Survey == null ? "null" : report.Survey.SurveyName);
    response = context.LoadProperty(report, "Survey");
    Console.WriteLine("LoadProperty Survey on Report returned {0}", response.StatusCode);
    Console.WriteLine("Found report on {0} for Survey {1}", report.Created, report.Survey.SurveyName);
    Console.ReadLine();
}

Conclusion

The potential of WCF data service is there. Just expose the data and let the clients filter and query what they need.

References

Demo project for download
Odata.org with all the documentation
Custom Data Service Providers on MSDN

Posted in Development | Tagged , , , , | Leave a comment

iPhoto/Aperture sync to iDevices

My home devices is mostly Apple o_O The integration is superb, but I fear the vendor lock-in.

Last week I tried to sync some pictures from Aperture to my iPad. Following the instructions from Apple it should be really simple. Just select the app/library you want to sync, check the albums and press sync. Unfortunately I was stuck on the first step.

iphotoaperture

I used to have iPhoto for managing my pictures but recently switched to Aperture. (I know it is going away) Thanks to Apple forward thinking Aperture could just use the iPhoto library. No need to migrate or import all my pictures. This turned out to be the issue.

To free up space (1.6Gb) on my SSD I removed the iPhoto app. Aperture worked like a charm and I didn’t even notice the sync in iTunes missing.

After some playing around with caches, small test libraries, recreated previews and library rebuilds I decided to bring back iPhoto. Since I don’t update to Yosemite (10.10) the App store wasn’t the place for me to get iPhoto. Luckily I use TimeMachine and could restore from there.

With iPhoto back in the Applications folder the option to sync with the library was back in iTunes. Check the albums and sync … just works.

TLDR

Make sure you have installed the app that created the library in order to sync to your iDevices with iTunes.

References

Sync photos on apple support
Aperture Library Not Showing Up on photoapps.expert

Posted in Tooling | Tagged , , , | Leave a comment