Atom Login Admin

Above the clouds

Google App Engine Pipeline API

written on Saturday,January 05,2013

Web Crawler of Lakshmi Projects that I have made ​​is using Google App Engine Pipeline API.
What is Pipeline API? There is Framework that introducing in project page;
which can be connected together to operate as Cascade,It also allows parallel processing.
Introduce simple way to run using unittest that according to sample code:

import unittest
import pipeline

class DivideWithRemainder(pipeline.Pipeline):

  output_names = ['remainder']

  def run(self, dividend, divisor):
    self.fill(self.outputs.remainder, dividend % divisor)
    return dividend // divisor

class EuclidGCD(pipeline.Pipeline):

  output_names = ['gcd']

  def run(self, a, b):
    a, b = max(a, b), min(a, b)
    if b == 0:
      self.fill(self.outputs.gcd, a)
    result = yield DivideWithRemainder(a, b)
    recurse = yield EuclidGCD(b, result.remainder)

class GCDTest(unittest.TestCase):
  def setUp(self):
    """Initialize temporary application variable."""

  def testGCD(self):
    """Test for GCD Pipeline"""
    stage = EuclidGCD(1071, 462)
    self.assertEqual(21, stage.outputs.gcd.value)

if __name__ == "__main__":
  • Run&Result
Ran 1 test in 0.276s


It is useful for concurrent jobs, Also support for Asynchronous operations as Waiting for user operations. which can to be more efficiency.
Google App Engine MapReduce is created by this framework.
Next, I will introduce Google App Engine MapReduce. coming soon:).


Add Comment

This entry was tagged #GAE