Splitting the Specs to a Separate Project with git Submodules?

Posted by rue, Thu Mar 13 01:47:00 UTC 2008

If we have a separate RubySpec repository which is included to Rubinius by means of git’s submodule support (mounted at spec/ruby/) there are two types of committers: those who work with RubySpec only and those who work with Rubinius only. The workflow for a RubySpec-only committer is very simple:

  1. Clone the main RubySpec to a local working copy.
  2. Make changes.
  3. Push changes to RubySpec mainline or make a patch if no commit bit.

That is the end for that committer. Eventually one of the Rubinius committers will pull in the changes to the Rubinius mainline. For a Rubinius committer wanting to make a change to specs, the process is a bit different. One note to make is that having RubySpecs as a separate project imposes one extra step to the normal workflow even if not making any changes to the specs. When first cloning the rbx repo, one must run `git submodule init && git submodule update` to pull in the specs—which is obviously needed to just RUN the specs. Then, if a `git pull` contains submodule changes, another update is usually required with `git submodule update`.

Rubinius Committers

If a person working with the Rubinius repo wants to make changes or add a new spec or whatever, they can of course choose to just work on a separate RubySpec checkout. If the committer does not have a commit bit to RubySpecs, they MUST use a separate repository. That works exactly as described above.

If, on the other hand, we have an individual who is both a Rubinius and RubySpecs committer and wants to work directly in the Rubinius checkout, then the workflow looks like this:

  1. `cd spec/ruby`
  2. `git checkout master`
  3. Make and commit changes. (Use branches if you like but come back to master.)
  4. Push changes to mainline RubySpecs (just `git push`.) <—THIS IS IMPORTANT
  5. `cd ../..`
  6. `git add spec/ruby` <—NO TRAILING /, that overwrites the submodule
  7. Commit.
  8. Push to mainline Rubinius.

At this point the updated submodule becomes available to others working with the Rubinius repo using `git pull && git submodule update`.

Full Example

Here is a long example of the entire process, covering some ground that TFM did not seem to do. Please send in any corrections and/or improvements.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381

  # We will work from scratch to illustrate the process
  $ cd ~/code/tmp

  # OK, first set up our submodule
  $ mkdir rubyspec
  $ cd rubyspec
  $ git init
   Initialized empty Git repository in .git/
  $ echo 'rubyspec' > README
  $ git add README
  $ git commit -a -m 'Initial import of rubyspec'
   Created initial commit 4c43e66: Initial import of rubyspec
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 README
  $ cd ..

  # OK, now set up the 'public' rubyspec repo (rubyspec.com or whatever)
  $ git clone --bare rubyspec rubyspec.git
   Initialized empty Git repository in /Users/user/code/tmp/rubyspec.git/

  # Next, we create the superproject, obviously the public rbx repo
  # already exists in reality.
  $ mkdir rbx
  $ cd rbx
  $ git init
   Initialized empty Git repository in .git/
  $ echo 'rbx' > README
  $ git add README
  $ git commit -a -m 'RBX import'
   Created initial commit 43b9035: RBX import
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 README
  $ cd ..

  # OK, same deal here, make a 'public' repo (git://git.rubini.us in reality)
  $ git clone --bare rbx rbx.git
   Initialized empty Git repository in /Users/user/code/tmp/rbx.git/

  # Now, let us get rid of the originals so we have a more realistic situation
  $ rm -rf rbx rubyspec
  $ ls
   rbx.git            rubyspec.git

  # Next we want to set up the submodule link, and we need a real rbx repo
  $ git clone rbx.git my_rbx
   Initialized empty Git repository in /Users/user/code/tmp/my_rbx/.git/
  $ cd my_rbx
  $ ls
   README
  # Now we associate the submodule (in reality we would use the URI, not local path)
  $ mkdir spec
  # We can 'mount' the submodule anywhere in the repo, in this case it goes in spec/ruby/
  $ git submodule add ~/code/tmp/rubyspec.git spec/ruby
   Initialized empty Git repository in /Users/user/code/tmp/my_rbx/spec/ruby/.git/
  $ ls -la
   total 16
   drwxr-xr-x  4 user  group  204 Mar 12 19:39 .
   drwxr-xr-x  7 user  group  238 Mar 12 19:37 ..
   drwxr-xr-x  7 user  group  374 Mar 12 19:39 .git
   -rw-r--r--  1 user  group   85 Mar 12 19:39 .gitmodules
   -rw-r--r--  1 user  group    4 Mar 12 19:37 README
   drwxr-xr-x  3 user  group  102 Mar 12 19:39 spec

  # .gitmodules has the tracking info
  $ cat .gitmodules
   [submodule "spec/ruby"]
           path = spec/ruby
           url = /Users/user/code/tmp/rubyspec.git
  # Our rubyspec repo should be in place now
  $ ls -la spec/ruby
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:39 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:39 ..
   drwxr-xr-x  7 user  group  374 Mar 12 19:39 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:39 README
  $ cat spec/ruby/README
   rubyspec

  # Now we need to record the submodule in the superproject
  $ git status
   On branch master
   Changes to be committed:
     (use "git reset HEAD <file>..." to unstage)

         new file:   .gitmodules
         new file:   spec/ruby

  $ git commit -m 'Imported rubyspec submodule'
   Created commit 0ae5cd3: Imported rubyspec submodule
    2 files changed, 4 insertions(+), 0 deletions(-)
    create mode 100644 .gitmodules
    create mode 160000 spec/ruby

  # And then record this over at the mainline
  $ git push
   Counting objects: 5, done.
   Compressing objects: 100% (3/3), done.
   Writing objects: 100% (4/4), 439 bytes, done.
   Total 4 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (4/4), done.
   To /Users/user/code/tmp/rbx.git
      43b9035..0ae5cd3  master -> master
  $ cd ..

  # Alright. Now, our test situation is that Joe just hacks on rubyspec and
  # Bob hacks on rubyspec as a part of rbx!

  # Let us create Bob's repo quick first:
  $ git clone rbx.git bobs_rbx
  $ cd bobs_rbx
  $ ls -la
   total 8
   drwxr-xr-x  4 user  group  204 Mar 12 19:47 .
   drwxr-xr-x  9 user  group  306 Mar 12 19:44 ..
   drwxr-xr-x  7 user  group  374 Mar 12 19:44 .git
   -rw-r--r--  1 user  group   85 Mar 12 19:44 .gitmodules
   -rw-r--r--  1 user  group    0 Mar 12 19:45 README
   drwxr-xr-x  3 user  group  102 Mar 12 19:44 spec

  # Now, as you can see, a pristine checkout of a superproject will not actually contain the submodule code
  $ ls -la spec/ruby
   total 0
   drwxr-xr-x  2 user  group   68 Mar 12 19:44 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:44 ..

  # We need to first init to properly link the modules and then update to pull the data in
  $ git submodule init
   Submodule 'spec/ruby' (/Users/user/code/tmp/rubyspec.git) registered for path 'spec/ruby'
  $ git submodule update
   Initialized empty Git repository in /Users/user/code/tmp/bobs_rbx/spec/ruby/.git/
   Submodule path 'spec/ruby': checked out '4c43e665381054478aa7c82da1d1d63b5c258e4b'
  $ ls -la spec/ruby
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README
  # OK, now it is there. Let us hop out of here for a bit.
  $ cd ..

  # Now, Joe is doing some work on his repo which is JUST the rubyspec submodule
  $ git clone rubyspec.git joes_spec
   Initialized empty Git repository in /Users/user/code/tmp/joes_spec/.git/
  $ cd joes_spec
  $ ls
   README
  # OK, Joe makes a change
  $ echo 'new spec' > spec.rb
  $ git add spec.rb
  $ git commit -m 'Added spec.rb'
   Created commit f380258: Added spec.rb
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 spec.rb
  # Joe pushes to mainline of rubyspec
  $ git push
   Counting objects: 4, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 290 bytes, done.
   Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rubyspec.git
      4c43e66..f380258  master -> master

  # Now, let us check in with Bob who has his clone of the rbx repo
  $ cd bobs_rbx

  # Bob does not have Joe's spec yet
  $ ls -la spec/ruby/
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README

  # Updating does not seem to work either!
  $ git submodule update
  $ ls -la spec/ruby/
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README

  # Did Joe's change get registered? Let us check..
  $ cd ..
  $ git clone rubyspec.git check_joes_change
   Initialized empty Git repository in /Users/user/code/tmp/check_joes_change/.git/
  $ ls check_joes_change/
   README  spec.rb
  # Yep, it is there! So what is going on?

  # Right, we need to update the mainline rbx. Technically Bob could actually do this
  # too but for now, let us pretend that some access control allows only me to make
  # that update. So back to my repo.
  $ cd my_rbx/
  $
  # Now, we need to grab the change first
  $ cd spec/ruby/
  $ ls
   README
  $ git status
   On branch master
   nothing to commit (working directory clean)
  $ git pull
   remote: Counting objects: 4, done.
   remote: Compressing objects: 100% (2/2)remote: , done.
   remote: Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   From /Users/user/code/tmp/rubyspec
      4c43e66..f380258  master     -> origin/master
   Updating 4c43e66..f380258
   Fast forward
    spec.rb |    1 +
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 spec.rb
  $ ls -la
   total 16
   drwxr-xr-x  3 user  group  170 Mar 12 19:54 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:39 ..
   drwxr-xr-x  7 user  group  442 Mar 12 19:54 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:39 README
   -rw-r--r--  1 user  group    9 Mar 12 19:54 spec.rb
  # OK, there it is!
  $ cd ../../
  # So, need to record this update in the superproject
  $ git status
   On branch master
   Changed but not updated:
     (use "git add <file>..." to update what will be committed)

         modified:   spec/ruby

   no changes added to commit (use "git add" and/or "git commit -a")
  $ git add spec/ruby
  $ git commit -m 'Updated our rubyspec checkout to newest version'
   Created commit 0d96517: Updated our rubyspec checkout to newest version
    1 files changed, 1 insertions(+), 1 deletions(-)
  # And then push this change to mainline
  $ git push
   Counting objects: 5, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 319 bytes, done.
   Total 3 (delta 1), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rbx.git
      0ae5cd3..0d96517  master -> master
  $ cd ..
  $ cd bobs_rbx/
  # NOW Bob should be able to access it
  $ git submodule update
  # No? How about this:
  $ git pull
   remote: Counting objects: 5, done.
   remote: Compressing objects: 100% (2/2), done.
   remote: Total 3 (delta 1), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   From /Users/user/code/tmp/rbx
      0ae5cd3..0d96517  master     -> origin/master
   Updating 0ae5cd3..0d96517
   README: needs update
   Fast forward
    spec/ruby |    2 +-
    1 files changed, 1 insertions(+), 1 deletions(-)
  # That by itself does not update yet
  $ ls -la spec/ruby
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README
  # Now we must update
  $ git submodule update
   Submodule path 'spec/ruby': checked out 'f38025812763dd8ee44beebcc0c8a3997c9aac0f'
  # And now it is there
  $ ls -la spec/ruby
   total 16
   drwxr-xr-x  3 user  group  170 Mar 12 19:57 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:57 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README
   -rw-r--r--  1 user  group    9 Mar 12 19:57 spec.rb
  # And there we go, properly updated now

  # Of course, Bob can make a few changes now too. Here is the only potential problem.
  # First, change something in rbx.
  $ echo 'RTFM' > INSTALL
  $ git add INSTALL
  $ git commit -m 'Installation instructions'
   Created commit c0e6667: Installation instructions
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 INSTALL
  $ git push
   Counting objects: 4, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 361 bytes, done.
   Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rbx.git
      0d96517..c0e6667  master -> master
  # That was normal. Now, what if Bob wants to make changes to rubyspec?
  # He can make the changes normally. If he has commit access, he can
  # just hop over and do so.
  $ cd spec/ruby
  $ ls
   README  spec.rb
  # One notable thing is that this is a detached by default to avoid some problems, so
  # Bob must go to a real branch.
  $ git status
   Not currently on any branch.
   nothing to commit (working directory clean)
  $ git checkout master
   Previous HEAD position was b292add... Added a spec
   Switched to branch "master"
  $ echo 'my spec' > spec2.rb
  $ git add spec2.rb
  $ git commit -m 'Added a spec'
   Created commit b1e7bdb: Added a spec
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 spec2.rb
  # Now Bob can just push it if he has access
  $ git push
   Counting objects: 4, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 313 bytes, done.
   Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rubyspec.git
      f380258..b1e7bdb  master -> master
  # As the data shows, we directly updated the rubyspec project
  $
  # Obviously, if Bob did not have commit rights, he would do the normal git-format-patch dance instead
  $
  # Now, the problem
  $ cd ../..
  $ git status
   On branch master
   Changed but not updated:
     (use "git add <file>..." to update what will be committed)

         modified:   spec/ruby

   no changes added to commit (use "git add" and/or "git commit -a")
  # This shows the change we made--and adding it is perfectly safe so long
  # as we do it correctly. This is incorrect:
  $ git add spec/ruby/
  $ git status
   On branch master
   Changes to be committed:
     (use "git reset HEAD <file>..." to unstage)

         deleted:    spec/ruby
         new file:   spec/ruby/README
         new file:   spec/ruby/spec.rb
         new file:   spec/ruby/spec2.rb

  $ git reset --hard HEAD
   HEAD is now at c0e6667... Installation instructions
  # OK, so that is not good. The trailing / has git thinking this is a new
  # path which overwrites the submodule. So, without:
  $ git add spec/ruby
  $ git status
   On branch master
   Changes to be committed:
     (use "git reset HEAD <file>..." to unstage)

         modified:   spec/ruby

  # So that worked fine. Now we can update mainline normally
  $ git commit -m 'New spec in rubyspec, updated here and pushed to rubyspec mainline'
   Created commit 3749f4b: New spec in rubyspec, updated here and pushed to rubyspec mainline
    1 files changed, 1 insertions(+), 1 deletions(-)
  $ git push
   Counting objects: 5, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 331 bytes, done.
   Total 3 (delta 1), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rbx.git
      c0e6667..3749f4b  master -> master
  $

Filed Under: rubinius | Tags:

Comments

Have your say

A name is required. You may use HTML in your comments.