Merge branch 'huge-repo-improve' into 'master'

Better support for huge repositories This MR contains next improvements to GitLab: * You can see repo commits count even if it is > 100k commits * Prevent 500 for commit page for large repos * Add notice about huge push over http to unicorn config * File action in satellites uses default 30 seconds timeout instead of old 10 seconds one * Show spinner when loading data for network graph

Merge branch 'huge-repo-improve' into 'master'
Better support for huge repositories This MR contains next improvements to GitLab: * You can see repo commits count even if it is > 100k commits * Prevent 500 for commit page for large repos * Add notice about huge push over http to unicorn config * File action in satellites uses default 30 seconds timeout instead of old 10 seconds one * Show spinner when loading data for network graph
Dmitriy Zaporozhets
2 parents 7edbee6c 61c2a2d3
Showing 9 changed files with 37 additions and 10 deletions Show diff stats
CHANGELOG
Gemfile
Gemfile.lock
app/controllers/projects/commit_controller.rb
app/helpers/application_helper.rb
app/models/repository.rb
app/views/projects/network/show.html.haml
config/unicorn.rb.example
lib/gitlab/satellite/files/file_action.rb
@@ -26,6 +26,11 @@ v 7.0.0
   - UI improvements for mobile devices
   - Improve diff rendering performance
   - Drag-n-drop for issues and merge requests between states at milestone page 
+  - Fix '0 commits' message for huge repositories on project home page
+  - Prevent 500 error page when visit commit page from large repo
+  - Add notice about huge push over http to unicorn config
+  - File action in satellites uses default 30 seconds timeout instead of old 10 seconds one
+  - Overall performance improvements
  
 v 6.9.2
   - Revert the commit that broke the LDAP user filter
@@ -790,4 +795,4 @@ v 0.8.0
   - stability
   - security fixes
   - increased test coverage
-  - email notification
 \ No newline at end of file
+  - email notification
@@ -30,7 +30,7 @@ gem &#39;omniauth-github&#39;
  
 # Extracting information from a git repository
 # Provide access to Gitlab::Git library
-gem "gitlab_git", '~> 5.8'
+gem "gitlab_git", '~> 6.0'
  
 # Ruby/Rack Git Smart-HTTP Server Handler
 gem 'gitlab-grack', '~> 2.0.0.pre', require: 'grack'
@@ -175,7 +175,7 @@ GEM
       mime-types (~> 1.19)
     gitlab_emoji (0.0.1.1)
       emoji (~> 1.0.1)
-    gitlab_git (5.9.0)
+    gitlab_git (6.0.0)
       activesupport (~> 4.0)
       charlock_holmes (~> 0.6)
       gitlab-grit (~> 2.6)
@@ -601,7 +601,7 @@ DEPENDENCIES
   gitlab-grack (~> 2.0.0.pre)
   gitlab-linguist (~> 3.0.0)
   gitlab_emoji (~> 0.0.1.1)
-  gitlab_git (~> 5.8)
+  gitlab_git (~> 6.0)
   gitlab_meta (= 6.0)
   gitlab_omniauth-ldap (= 1.0.4)
   gollum-lib (~> 3.0.0)
@@ -12,7 +12,12 @@ class Projects::CommitController &lt; Projects::ApplicationController
     return git_not_found! unless @commit
  
     @line_notes = project.notes.for_commit_id(commit.id).inline
-    @branches = project.repository.branch_names_contains(commit.id)
+
+    @branches = begin
+                  project.repository.branch_names_contains(commit.id)
+                rescue Grit::Git::GitTimeout
+                  []
+                end
  
     begin
       @suppress_diff = true if commit.diff_suppress? && !params[:force_show_diff]
@@ -226,8 +226,11 @@ module ApplicationHelper
     GitHub::Markup.render(file_name, file_content).html_safe
   end
  
-  def spinner(text = nil)
-    content_tag :div, class: 'loading hide' do
+  def spinner(text = nil, visible = false)
+    css_class = "loading"
+    css_class << " hide" unless visible
+
+    content_tag :div, class: css_class do
       content_tag(:i, nil, class: 'icon-spinner icon-spin') + text
     end
   end
@@ -106,7 +106,7 @@ class Repository
   def commit_count
     Rails.cache.fetch(cache_key(:commit_count)) do
       begin
-        raw_repository.raw.commit_count(self.root_ref)
+        raw_repository.commit_count(self.root_ref)
       rescue
         0
       end
@@ -3,7 +3,7 @@
   .tip
     You can move around the graph by using the arrow keys.
   .network-graph
-    = spinner
+    = spinner nil, true
  
 :javascript
   new Network({
@@ -34,6 +34,20 @@ listen &quot;/home/git/gitlab/tmp/sockets/gitlab.socket&quot;, :backlog =&gt; 64
 listen "127.0.0.1:8080", :tcp_nopush => true
  
 # nuke workers after 30 seconds instead of 60 seconds (the default)
+#
+# NOTICE: git push over http depends on this value. 
+# If you want be able to push huge amount of data to git repository over http 
+# you will have to increase this value too. 
+# 
+# Example of output if you try to push 1GB repo to GitLab over http.
+#   -> git push http://gitlab.... master
+#
+#   error: RPC failed; result=18, HTTP code = 200
+#   fatal: The remote end hung up unexpectedly
+#   fatal: The remote end hung up unexpectedly
+#
+# For more information see http://stackoverflow.com/a/21682112/752049
+#
 timeout 30
  
 # feel free to point this anywhere accessible on the filesystem
@@ -4,7 +4,7 @@ module Gitlab
       attr_accessor :file_path, :ref
  
       def initialize(user, project, ref, file_path)
-        super user, project, git_timeout: 10.seconds
+        super user, project
         @file_path = file_path
         @ref = ref
       end
...	...	@@ -26,6 +26,11 @@ v 7.0.0
26	26	- UI improvements for mobile devices
27	27	- Improve diff rendering performance
28	28	- Drag-n-drop for issues and merge requests between states at milestone page
	29	+ - Fix '0 commits' message for huge repositories on project home page
	30	+ - Prevent 500 error page when visit commit page from large repo
	31	+ - Add notice about huge push over http to unicorn config
	32	+ - File action in satellites uses default 30 seconds timeout instead of old 10 seconds one
	33	+ - Overall performance improvements
29	34
30	35	v 6.9.2
31	36	- Revert the commit that broke the LDAP user filter
...	...	@@ -790,4 +795,4 @@ v 0.8.0
790	795	- stability
791	796	- security fixes
792	797	- increased test coverage
793		- - email notification
794	798	\ No newline at end of file
	799	+ - email notification
...	...
...	...	@@ -30,7 +30,7 @@ gem 'omniauth-github'
30	30
31	31	# Extracting information from a git repository
32	32	# Provide access to Gitlab::Git library
33		-gem "gitlab_git", '~> 5.8'
	33	+gem "gitlab_git", '~> 6.0'
34	34
35	35	# Ruby/Rack Git Smart-HTTP Server Handler
36	36	gem 'gitlab-grack', '~> 2.0.0.pre', require: 'grack'
...	...
...	...	@@ -175,7 +175,7 @@ GEM
175	175	mime-types (~> 1.19)
176	176	gitlab_emoji (0.0.1.1)
177	177	emoji (~> 1.0.1)
178		- gitlab_git (5.9.0)
	178	+ gitlab_git (6.0.0)
179	179	activesupport (~> 4.0)
180	180	charlock_holmes (~> 0.6)
181	181	gitlab-grit (~> 2.6)
...	...	@@ -601,7 +601,7 @@ DEPENDENCIES
601	601	gitlab-grack (~> 2.0.0.pre)
602	602	gitlab-linguist (~> 3.0.0)
603	603	gitlab_emoji (~> 0.0.1.1)
604		- gitlab_git (~> 5.8)
	604	+ gitlab_git (~> 6.0)
605	605	gitlab_meta (= 6.0)
606	606	gitlab_omniauth-ldap (= 1.0.4)
607	607	gollum-lib (~> 3.0.0)
...	...
...	...	@@ -12,7 +12,12 @@ class Projects::CommitController < Projects::ApplicationController
12	12	return git_not_found! unless @commit
13	13
14	14	@line_notes = project.notes.for_commit_id(commit.id).inline
15		- @branches = project.repository.branch_names_contains(commit.id)
	15	+
	16	+ @branches = begin
	17	+ project.repository.branch_names_contains(commit.id)
	18	+ rescue Grit::Git::GitTimeout
	19	+ []
	20	+ end
16	21
17	22	begin
18	23	@suppress_diff = true if commit.diff_suppress? && !params[:force_show_diff]
...	...
...	...	@@ -226,8 +226,11 @@ module ApplicationHelper
226	226	GitHub::Markup.render(file_name, file_content).html_safe
227	227	end
228	228
229		- def spinner(text = nil)
230		- content_tag :div, class: 'loading hide' do
	229	+ def spinner(text = nil, visible = false)
	230	+ css_class = "loading"
	231	+ css_class << " hide" unless visible
	232	+
	233	+ content_tag :div, class: css_class do
231	234	content_tag(:i, nil, class: 'icon-spinner icon-spin') + text
232	235	end
233	236	end
...	...
...	...	@@ -106,7 +106,7 @@ class Repository
106	106	def commit_count
107	107	Rails.cache.fetch(cache_key(:commit_count)) do
108	108	begin
109		- raw_repository.raw.commit_count(self.root_ref)
	109	+ raw_repository.commit_count(self.root_ref)
110	110	rescue
111	111	0
112	112	end
...	...
...	...	@@ -3,7 +3,7 @@
3	3	.tip
4	4	You can move around the graph by using the arrow keys.
5	5	.network-graph
6		- = spinner
	6	+ = spinner nil, true
7	7
8	8	:javascript
9	9	new Network({
...	...
...	...	@@ -34,6 +34,20 @@ listen "/home/git/gitlab/tmp/sockets/gitlab.socket", :backlog => 64
34	34	listen "127.0.0.1:8080", :tcp_nopush => true
35	35
36	36	# nuke workers after 30 seconds instead of 60 seconds (the default)
	37	+#
	38	+# NOTICE: git push over http depends on this value.
	39	+# If you want be able to push huge amount of data to git repository over http
	40	+# you will have to increase this value too.
	41	+#
	42	+# Example of output if you try to push 1GB repo to GitLab over http.
	43	+# -> git push http://gitlab.... master
	44	+#
	45	+# error: RPC failed; result=18, HTTP code = 200
	46	+# fatal: The remote end hung up unexpectedly
	47	+# fatal: The remote end hung up unexpectedly
	48	+#
	49	+# For more information see http://stackoverflow.com/a/21682112/752049
	50	+#
37	51	timeout 30
38	52
39	53	# feel free to point this anywhere accessible on the filesystem
...	...
...	...	@@ -4,7 +4,7 @@ module Gitlab
4	4	attr_accessor :file_path, :ref
5	5
6	6	def initialize(user, project, ref, file_path)
7		- super user, project, git_timeout: 10.seconds
	7	+ super user, project
8	8	@file_path = file_path
9	9	@ref = ref
10	10	end
...	...