Mode II fracture initiation and propagation plays an important role under certain loading conditions in rock fracture mechanics. Under pure tensile, pure shear, tension- and compression-shear loading, the maximum Mode I stress intensity factor, , is always larger than the maximum Mode II stress intensity factor, . For brittle materials, Mode I fracture toughness, KIC, is usually smaller than Mode II fracture toughness, KIIC. Therefore, reaches KIC before reaches KIIC, which inevitably leads to Mode I fracture. Due to inexistence of Mode II fracture under pure shear, tension- and compression-shear loading, classical mixed mode fracture criteria can only predict Mode I fracture but not Mode II fracture. A new mixed mode fracture criterion has been established for predicting Mode I or Mode II fracture of brittle materials. It is based on the examination of Mode I and Mode II stress intensity factors on the arbitrary plane θ,KI(θ) and KII(θ), varying with θ(−180°⩽θ⩽+180°), no matter what kind of loading condition is applied. Mode I fracture occurs when or and at θIC. Mode II fracture occurs when and at θIIC. The validity of the new criterion is demonstrated by experimental results of shear-box testing.Shear-box test of cubic specimen is a potential method for determining Mode II fracture toughness KIIC of rock since it can create a favorable condition for Mode II fracture, i.e. is always 2–3 times larger than and reaches KIIC before reaches KIC. The size effect on KIIC for single- and double-notched specimens has been studied for different specimen thickness B, dimensionless notch length a/W (or 2a/W) and notch inclination angle α. The test results show that KIIC decreases as B increases and becomes a constant when B is equal to or larger than W for both the single- and double-notched specimens. When a/W (or 2a/W) increases, KIIC decreases and approaches a limit. The α has a minor effect on KIIC when α is within 65–75°. Specimen dimensions for obtaining a reliable and reproducible value of KIIC under shear-box testing are presented. Numerical results demonstrate that under the shear-box loading condition, tensile stress around the notch tip can be effectively restrained by the compressive loading. At peak load, the maximum normal stress is smaller than the tensile strength of rock, while the maximum shear stress is larger than the shear strength in the presence of compressive stress, which results in shear failure.