Merge Request #38
Gitlab data
Make sure you have gitlab plugin active and properly configurated PROXIED_APPS: gitlab: upstream: 'http://localhost:8090/gitlab/' private_token: ''
You'll need a valid administrator private_token to validate import_proxy_data It might take a while if the upstream server is external depending on the delay
-
Signed-off-by: Gustavo Jaruga <darksshades@gmail.com> Signed-off-by: Rodrigo Siqueira <rodrigosiqueiramelo@gmail.com>
-
Sign-off-by: Rodrigo Siqueira <rodrigosiqueiramelo@gmail.com>
-
Signed-off-by: Gustavo Jaruga <darksshades@gmail.com> Signed-off-by: Rodrigo Siqueira de Melo <rodrigosiqueiramelo@gmail.com>
-
Signed-off-by: Gustavo Jaruga <darksshades@gmail.com>
- 8 of 12 commits displayed. Click here to show all
-
Why such a big block of code is inside a try for KeyError?
-
You highlighted the wrong code. Every attribution of "value = element[somevalue]" in this function has a chance of not existing and give a KeyError. This functions tries to generalize the attributions of the GitlabData expecting the the json data to be the same name, which happens 80% of the time. Others specific attributions are made outside this funcion.
-
Isn't it a bit dangerous? If the server doesn't send the parameter you are expecting you might end-up in an infinite loop. Isn't there a why to calculate the number of pages before hand?
-
The same applies for all loops while True loops in this file.
-
GitlabAPI doesn't provide any way to know how many pages there are... But when you input a nonexistent page or the parser fails, it returns an empty list and breaks the While.
-
What about GitlabProject(**element)? You would need to thread element to be in the right format but it might be a good idea. Perhaps a class method
GitlabProject.from_api. -
We didn't understand.... the function fill_object_data is used to fill the generic fields of GitlabProject, and any code bellow that are used to fill the specific ones.
-
Great progress here! What about the peer reviews?
| 32 | + def get_json_data(self, api_url, page, pages=1000): | |
| 33 | + url = self.get_request_url(api_url, per_page=pages, | |
| 34 | + page=page) | |
| 35 | + | |
| 36 | + try: | |
| 37 | + data = urllib2.urlopen(url, timeout=10) | |
| 38 | + json_data = json.load(data) | |
| 39 | + except urllib2.URLError: | |
| 40 | + LOGGER.exception("Connection timeout: " + url) | |
| 41 | + json_data = [] | |
| 42 | + | |
| 43 | + return json_data | |
| 44 | + | |
| 45 | + def fill_object_data(self, element, _object): | |
| 46 | + for field in _object._meta.fields: | |
| 47 | + try: | |
| 2 |
|
|
| 63 | + continue | |
| 64 | + | |
| 65 | + return _object | |
| 66 | + | |
| 67 | + def fetch_projects(self): | |
| 29 | 68 | page = 1 |
| 30 | 69 | projects = [] |
| 31 | 70 | |
| 32 | - # Iterates throughout all projects pages | |
| 33 | - while(True): | |
| 34 | - url = self.get_request_url('/api/v3/projects/all', | |
| 35 | - per_page=100, | |
| 36 | - page=page) | |
| 37 | - data = urllib2.urlopen(url) | |
| 38 | - json_data = json.load(data) | |
| 71 | + while True: | |
| 3 |
|
|
| 41 | 76 | break |
| 42 | 77 | |
| 43 | - page = page + 1 | |
| 44 | - | |
| 45 | 78 | for element in json_data: |
| 46 | 79 | project = GitlabProject() |
| 47 | - | |
| 48 | - for field in GitlabProject._meta.fields: | |
| 49 | - if isinstance(field, DateTimeField): | |
| 50 | - value = parse(element[field.name]) | |
| 51 | - else: | |
| 52 | - value = element[field.name] | |
| 53 | - | |
| 54 | - setattr(project, field.name, value) | |
| 55 | - | |
| 80 | + self.fill_object_data(element, project) | |
| 2 |
|
|