Nithin Nithin - 3 months ago 12x
Python Question

Cyclic dependencies and interfaces in golang

I am a long time python developer. I was trying out go, converting an existing python app to go. It is modular and works really well for me.

Upon creating the same structure in go, I seem to land in cyclic import errors, a lot more than I want to. Never had any import problems in python. I never even had to use import aliases. So I may have had some cyclic imports which were not evident in python. I actually find that strange.

Anyways, I am lost, trying to fix these in go. I have read that interfaces can be used to avoid cyclic dependencies. But I don't understand how. I didn't find any examples on this either. Can somebody help me on this?


The current python application structure is as follows:


  • settings

    • -> contains main routes depends on app1/, app2/ etc

    • -> function like connect() which opens db session

    • -> general constants

  • apps

    • app1

      • -> url handler functions

      • -> app specific database functions depends on settings/

      • -> app specific routes

      • ...

    • app2

      • -> url handler functions

      • -> app specific database functions depends on settings/

      • -> app specific routes

      • ...

settings/ has generic functions like connect() which opens a db session. So an app in the apps package calls database.connect() and a db session is opened.

The same is the case with settings/ it has functions that allow apps to add their sub-routes to the main route object.

The settings package is more about functions than data/constants. This contains code that is used by apps in the apps package, that would otherwise have to be duplicated in all the apps. So if I need to change the router class, for instance, I just have to change settings/ and the apps will continue to work with no modifications.


The short version:

  • Write config functions for hooking packages up to each other at run time rather than compile time. Instead of routes importing all the packages that define routes, it can export routes.Register, which main (or code in each app) can call.

  • As a rule, split a package up when each piece could be useful on its own. If two pieces of functionality are really intimately related, you don't have to split them into packages at all; you can organize with multiple files or types instead. Go's net/http is a big package, for instance, and understandably so.

More specifically:

  • Move reusable code 'down' into lower-level packages untangled from your particular use case. If you have a package page containing both logic for your content management system and all-purpose HTML-manipulation code, consider moving the HTML stuff "down" to a package html so you can use it without importing unrelated content management stuff.

  • Break up grab-bag packages (utils, tools) by topic or dependency. Otherwise you can end up importing a huge utils package (and taking on all its dependencies) for one or two pieces of functionality (that wouldn't have so many dependencies if separated out).

  • Pass around basic types and interface values. If you're depending on a package for just a type name, maybe you can avoid that. Maybe some code handling a []Page can get instead use a []string of filenames or a []int of IDs or some more general interface (sql.Rows) instead.

Related to the these points, Ben Johnson gave a lightning talk at GopherCon 2016. He suggests breaking up packages by dependency, and defining one package that just has interfaces and data types, without any but the most trivial functionality (and as a result few to no dependencies); in his words it defines the "language" of your app.

Here, I'd rearrange things so the router doesn't need to include the routes: instead, each app package calls a router.Register() method. This is what the Gorilla web toolkit's mux package does. Your routes, database, and constants packages sound like low-level pieces that should be imported by your app code and not import it.

Generally, try to build your app in layers. Your higher-layer, use-case-specific app code should import lower-layer, more fundamental tools, and never the other way around. Here are some more thoughts:

  • Packages are for separating independently usable bits of functionality; you don't need to split one off whenever a source file gets large. Unlike in, say, Python or Java, in Go one can split and combine and rearrange files completely independent of the package structure, so you can break up huge files without breaking up packages.

    The standard library's net/http is about 7k lines (counting comments/blanks but not tests). Internally, it's split into many smaller files and types. But it's one package, I think 'cause there was no reason users would want, say, just cookie handling on its own. On the other hand, net and net/url are separate because they have uses outside HTTP.

    It's great if you can push "down" utilities into libraries that are independent and feel like their own polished products, or cleanly layer your application itself (e.g., UI sits atop an API sits atop some core functionality and data models). Likewise "horizontal" separation may help you hold the app in your head (e.g., the UI layer breaks up into user account management, the application core, and administrative tools, or something finer-grained than that). But, the core point is, you're free to split or not as works for you.

  • Use Register or other runtime config methods to keep your general tools (like URL routing or DB access code) from needing to import your app code. Instead of your router looking at app1.Routes, app2.Routes, etc., you have your apps packages import router and register with it in their func init()s.

    Or, if you'd rather register routes from one package, you could make a myapp/routes package that imports router and all your views and calls router.Register. Point is, the router itself is all-purpose code that needn't import your application's views.

    Some ways to put together config APIs:

    • Pass app behavior via interfaces or funcs: http can be passed custom implementations of Handler (of course) but also CookieJar or File. text/template and html/template can accept functions to be accessible from templates (in a FuncMap).

    • Export shortcut functions from your package if appropriate: In http, callers can either make and separately configure some http.Server objects, or call http.ListenAndServe(...) that uses a global Server. That gives you a nice design--everything's in an object and callers can create multiple Servers in a process and such--but it also offers a lazy way to configure in the simple single-server case.

    • If you have to, just duct-tape it: You don't have to limit yourself to super-elegant config systems if you can't fit one to your app: should your app have package "myapp/conf" with a global var Conf map[string]interface{}, I won't judge. My one warning would be that this ties every conf-importing package to your app: if some might otherwise be reusable outside your application, maybe you can find a better way to configure them.

Those two are maybe the key principles, but a couple of specific cases/tactical thoughts:

  • Separate fundamental tasks from app-dependent ones. One app I work on in another language has a "utils" module mixing general tasks (e.g., formatting datetimes or working with HTML) with app-specific stuff (that depends on the user schema, etc.). But the users package imports the utils, creating a cycle. If I were porting to Go, I'd move the user-dependent utils "up" out of the utils module, maybe to live with the user code or even above it.

  • Consider breaking up grab-bag packages. Slightly enlarging on the last point: if two pieces of functionality are independent (that is, things still work if you move some code to another package) and unrelated from the user's perspective, they're candidates to be separated into two packages. Sometimes the bundling is harmless, but other times it leads to extra dependencies, or a less generic package name would just make clearer code. So my utils above might be broken up by topic or dependency (e.g., strutil, dbutil, etc.). If you wind up with lots of packages this way, we've got goimports to help manage them.

  • Replace import-requiring object types in APIs with basic types and interfaces. Say two entities in your app have a many-to-many relationship like Users and Groups. If they live in different packages (a big 'if'), you can't have both u.Groups() returning a []group.Group and g.Users() returning []user.User because that requires the packages to import each other.

    However, you could change one or both of those return, say, a []uint of IDs or a sql.Rows or some other interface you can get to without importing a specific object type. Depending on your use case, types like User and Group might be so intimately related that it's better just to put them in one package, but if you decide they should be distinct, this is a way.

Thanks for the detailed question and followup.