How to understand VertexTextureCoordinates in Texture?

This happens because texturing is done triangle by triangle. Polygons with more sides are broken down into triangles, and each triangle is textured individually. I believe your example is effectively equivalent to

pic = ExampleData[{"TestImage", "Mandrill"}];

Graphics[{Texture[pic], EdgeForm[Black], 
  Polygon[{{0, 0}, {1, 0}, {2, 2}}, 
   VertexTextureCoordinates -> {{0, 0}, {2, 0}, {2, 2}}], 
  Polygon[{{2, 2}, {0.5, 2.5}, {0, 0}}, 
   VertexTextureCoordinates -> {{2, 2}, {0, 2}, {0, 0}}]}]

Note first : it is recommended to read this answer after the two other ones (from Szabolcs and matheorem)

Here is a tool intended to explore how VertexTextureCoordinates works.

The texture is the image :

The evaluation of the code at the end of the answer returns the following :

A1,A2,A3 ..., B1,B2,B3 can be independently moved with the mouse (on both sides).

• the polygon A1,A2,A3... shows what piece of the texture is extracted.

• the polygon B1,B2,B3... shows where goes the piece

The relevant corresponding part of the code is :

Texture[the image] ...
Polygon[{B1,B2,B3...},VertexTextureCoordinates-> {A1,A2,A3...}]

Note :

• If one adds a locator (with the button "add locator") and move the new locator on the left side, nothing happens on the right side. This let think that the update in the right side is not real time. This is not true : simply move the new locator on the right side, then after the corresponding locator on the left side and see...

• The Button in the middle simply does B1=A1;B2=A2;B3=A3... . See the effect on a distorded image.

• The code shows 4 images simply to illustrate a wrapping effect on the texture when the scaled coordinates are not in the interval [0,1]

It's really interesting to watch how the triangulation changes when the locators on the right side are moved, for example by taking the example "big hat" (simply press the button) and moving B2 upwards.

Code

myHatching=RegionPlot[Rectangle[{0,0},{512,512}],Mesh-> 30,
MeshFunctions->{#1-#2 &, #1+#2 & },PlotStyle->Opacity[0],MeshStyle->
{Directive[AbsoluteThickness[2],Black],Directive[AbsoluteThickness[3],Red]}];
img00=Rasterize[Show[Lighter @ ExampleData[{"TestImage", "Lena"}],myHatching,ImageSize-> 512]];
img00X4=ImageAssemble[{{img00,img00},{img00,img00}}];

ptsA= ptsB= {{250,250},{375,150},{500,250},{500,500},{250,500}};

Row[{
LocatorPane[
   Dynamic[ptsA],
   Show[img00X4,Frame-> True,ImageSize-> 500,Epilog-> Dynamic[
      {
      Thick,
      Line[Append[ptsA,ptsA[[1]]]],
      MapIndexed[Style[Text["A"<>ToString[#2 //First] ,#+{15,15}],FontSize->14,Bold,Black]&,ptsA]
      }], DataRange-> {{0,2},{0,2}}],
   Appearance->Graphics[{Black,Disk[]},ImageSize->7]
],
Button[" -->\ndo :\n B1=A1\n B2=A2\n ...\n Bn=An\n -->",ptsB=ptsA],
LocatorPane[
   Dynamic[ptsB],
   Dynamic[Graphics[
      {
         Texture[img00],
         Polygon[ptsB,VertexTextureCoordinates-> (ptsA/512.)],
         MapIndexed[Style[Text["B"<>ToString[#2 //First],#+{15,15}],FontSize->14,Bold]&,ptsB]
      },
      PlotRange->{{0,1024},{0,1024}},Frame-> True,ImageSize-> 500]],
   Appearance->Graphics[{Black,Disk[]},ImageSize->7]]
},"   "] //Style[#,ImageSizeMultipliers-> {1,1}]&

Button[
"add locator",
ptsB=Insert[ptsB,(ptsB[[1]]+ptsB[[2]])/2.,2];
ptsA=Insert[ptsA,(ptsA[[1]]+ptsA[[2]])/2.,2]]

Button["sample 1 : big hat",
ptsB={{87.,249.},{375.,150.},{500.,250.},{432.,457.},{21.,659.}};
ptsA={{87.,249.},{375.,150.},{500.,250.},{432.,457.},{137.,481.}};] 

(It's quick and dirty code, ie not a reference for programmer.)

the number of coordinates in VertexTextureCoordinates should be the same as the number of vertex in polygon.

As an example

Graphics[{Texture[pic], EdgeForm[Black],
  Polygon[{{0, 0}, {1, 0}, {2, 2}, {0.5, 2.5}},
   VertexTextureCoordinates -> {{0, 0}, {2, 0}, {2, 2}, {0, 2}}]}]

The texture process is actually means to think the pic as stretchable membrane and do four steps:

1. grab {0,0} point (the lower left point) of the pic and stick it on polygon's {0,0} vertex
2. grab {2,0} point of the pic and stick it on polygon's {1,0} vertex. But since {2,0} is actually outside the pic, we should really think we have extended periodic array of pic, and the coordinates is as follows But now we only have stick two points and only got line.
3. grab {2,2} point of the pic and stick it on polygon's {2,2} vertex. Now we have stick three points, and they form a triangle, and we got a triangle area textured with pic, and there is still one triangle left.
4. grab {0,2} point of the pic and stick it on polygon's {0.5, 2.5} vertex. Now we the remaining triangle textured. and got

Also note, unequal points will be filled with {0,0}. That is

VertexTextureCoordinates -> {{0, 0}, {2, 0}, {2, 2}}

is equivalent to

VertexTextureCoordinates -> {{0, 0}, {2, 0}, {2, 2},{0,0}}