More cloud streaming

In Stream content to Google Cloud Storage I covered how you could stream content (in this case a video file) from a hosted url to an object on cloud storage in preparation for labelling it with the Video Intelligence API.
Let’s see how generalize that a bit so it can be used for all reading and writing of content to and from Cloud Storage.

Why streaming

Using streaming abstracts away the size of the content you are trying to stream, and also generalizes whether it is content in memory in your app, or some content coming from a file, a request or some other piped resource. So I’ll be using streaming for all interactions with cloud storage.

General usage

Let’s say we have some content to get to storage, we want to be able to simply express it like this.

Storagestream

And here’s the streamContent function

There’s a couple of dependencies there – (you can get this from

And the next is a couple of functions to create a writestream attached to a cloud blob. The credenentials are a service account and bucket name that provide access to the cloud storage platform as discussed in Stream content to Google Cloud Storage

and that’s all that’s needed to stream content to cloud storage.

So let’s say your content is being piped to your app from an http request or the files system or some other process.  Since the example above converts the string to a stream then pipes it to the cloud storage writestream, it’s very much the same process. Here’s the Stream content to Google Cloud Storage example, using the same functions as above. All that’s required is the name (in this case the url of the video file to be streamed, and optionally the mimeType) from which a writestream can be constructed, and then piping the content to be downloaded directly to that writestream.

Reading content

That resolves to a one liner like this

And here’s the function

Code

The storagestream code is on github. You’ll need to construct your own secrets file to provide credentials and bucket information.